In the field of enterprise computer networks, e.g. as sketched in FIG. 1A by an enterprise's intranet 10, today's computer industry is moving toward fast, packetized, serial input/output (I/O) bus architectures, in which computing hosts like the exemplary database server 12 and peripherals like an Internet mail server 14 are linked by a switching network, commonly referred to as a switching fabric. A number of architectures of this type have been proposed, culminating in the “InfiniBand™” (IB) architecture, which has been advanced by a consortium led by a group of industry leaders. The IB architecture is described in detail in the InfiniBand™ Architecture Specification, which is available from the InfiniBand™-Trade Association at www.infinibandta.org and is incorporated herein by reference.
InfiniBand™ technology connects the hardware of two channel adapters 16, further abbreviated herein as CA, by using Queue Pairs further abbreviated herein as QPs. Those QPs have associated with them a Send Queue and a Receive Queue. The QPs are set up by software. So each application can have multiple QPs for different purposes. Each QP has associated with it a Queue Pair Context further abbreviated herein as QPC, which contains information about the type of the QP, e.g. whether it concerns a reliable or an unreliable connection.
If an application wants to use a QP, it has to send a Work Request, further abbreviated herein as WR, to the Channel Adapter (CA). A work request gets translated into an InfiniBand™-defined Work Queue Element further abbreviated herein as WQE, and is made available on the send or receive queue of the QP. The list of WQEs, which belong to a given QP, is stored in the QPC. This is true not only for the send side, but for the receive side as well, except in cases of Remote Direct memory Access (RDMA). The WQEs contain information, where to store received data, in the system memory of the receiving side computer.
FIG. 1B shows a block diagram illustrating a physical overview of a system, which contains an InfiniBand™ Channel Adapter 16.
The system consists of one or more processors 15 and a system memory 18. Within the memory there is section 11 containing outstanding Work Requests and a data section 13, which is organized divided into several Data Segments.
The processor(s) and the memory 18 are connected to a System Interconnect Structure 19. This System Interconnect is implemented in form of an adequate bus structure and has connection to the CA 16.
Within the CA 16 there are one ore more Packet Receive Processor(s) 17 PRP, which are connected to the physical port 9 of the CA 16. The task of the Packet Receive Processors 17 (PRPs) is to analyze incoming packets and store them into the system memory.
FIG. 2 is to give a short overview over the internal structure and the basic functionality of the receive side of a prior art Channel Adapter 16.
A new packet comes in over the physical link 20. According to the InfiniBand™ Specification the link can have work on different speeds.
First the packet is stored in Virtual Lane In Manager (VLInMager) 22. This unit is needed to reduce backpressure to the link. It may be basically assumed to be a large data array.
Over a predetermined dispatch algorithm the packets are transmitted to the PRPs 17. They analyze the packet and fetch some data over a dedicated logic, referred herein as Queue Pair Context Manager (QPCM) 24.
To handle the packet correctly, some data has to be fetched over the System Interconnect Logic 19.
With reference to FIGS. 3, 4 and 5 the data structures as used in prior art are briefly described in order to give a full understanding to the skilled reader.
FIG. 3 shows an InfiniBand™ packet. It consists of an Header 30 followed by Data 32. At the end of a packet there is an CRC section 34.
FIG. 4 shows a QP Context. It contains a lot of context data 40 concerning the QP, e.g., telling if the connection type is reliable or unreliable. Additionally, it contains pointers 42 A,B, . . . to the multiple Work Queue Elements (WQEs), which belong to that QP.
FIG. 5 shows a high level overview of a prior art WQE 50. It contains some quantity of Meta data 52 concerning the Work Request. Additionally there are pointers 54 A, 54B, 54C to specific Data Sections in the system memory.
These structures are described in detail in the InfiniBand™ Specification.
With reference to FIG. 6 further details of the prior art data flow are described. After a packet has arrived it is stored in a data array 60, which belongs to before-mentioned VLIn Manager 22. With a given dispatch algorithm a packet 62 is presented to a PRP 17.
The PRP requests from the QPCM 24 the QPC of the QP Number denoted as QP#, which belongs to and identifies the packet. If the context is in a cache 64 of the QPCM 24, it is presented immediately to the PRP, see arrow 66. Else it is requested and fetched, see arrows 68, from memory using the System Interconnect 19.
After the PRP has received the context, it fetches (requests and receives) the WQE 50 from memory using the System Interconnect, see arrows 69.
Approaching now the problem underlying the present invention, the sum of all WQE and QPCs, which belong to a given CA 16, can be too big to be stored on the CA itself. So, a well known solution is to store the WQE and QPCs in the System Memory 18 and fetch them from the memory via any system interconnect means 19, ie, the before-mentioned bus system, when needed. A prior art improvement of that basic approach is to use caches located on the CA.
The initial problem of “outsourcing” the WQEs and QPCs to the systems memory 18 into a work request (WR) queue 11 consists in the fact, that it needs considerable time to fetch them from there to the channel adapter's chip.
After a packet arrives at its chip the CA has to find out to which QP that packet belongs by analyzing the header.
Then the CA has to fetch the QPC from the system memory. After analyzing the QPC the CA can start fetching the right WQE.
During that time the packet disadvantageously remains unused in the chip and occupies computing resources, as e.g. processor 15 and memory 18.
Thus, this prior art outsourcing of WQEs and QPCs to the system memory disadvantageously costs a lot of performance.
In order to reduce that impact one can implement more packet receive processors 17 (PRPs), which are working on the packets. Working on more packets in parallel reduces the loss of performance.
But this requires increasing the chip size of the channel adapter, which would make the chip disadvantageously more expensive.
A straight-forward solution to that would be the use of caches. That means that a certain amount of QPCs and WQEs could be stored in the chip cache and thus on-chip. Once the CA would have fetched a QPC or WQE it would stay in the CA, because it is most likely that there would be more packets in a row for the same QP. So this information could be reused.
Since, however, the cache size is not infinite, QPCs and WQEs would have to be deleted from the cache to be able to store new QPCs and WQEs which would be needed. That means that the original problem to fetch the data out of the system memory 18 is not really solved, but instead it is just reduced. Further, generally, cache size is not for free. It costs a lot of chip size, which makes the chip disadvantageously much more expensive.